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ABSTRACT 

Traditional instructional materials such as textbooks contain significant educational content, but 
thenavigational mechanisms to access that content are limited and, more importantly, not designed 
with learning in mind. To address this gap, we present the Adaptive Map, a novel organization and 
navigation tool designed to help students better understand large collections of information such as 
those found in textbooks. The tool promotes conceptual understanding by leveraging expert-generated 
concept maps, which have been shown to improve learning. In addition, to alleviate problems associ¬ 
ated with visually processing large-scale concept maps, our design integrated information visualization 
techniques to increase usability and facilitate processing. To evaluate the approach, a prototype was 
developed and tested in an engineering statics class. A multiple case study methodology was used to 
explore how students used the tool. Results show that the tool promoted conceptual understanding 
by facilitating advance organizer usage and students’ exploration of relevant prior content. 
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INTRODUCTION 

Within engineering education practice, centralized sources of information such as textbooks that 
students can access outside the classroom are commonly used. These sources of information have 
been shown to support out-of-the-classroom learning in such activities as studying for exams or 
completing homework assignments (Taraban et al. 2004), but only limited research has examined 
how students use these tools and what effect the design of these tools has on learning (Lee et al. 
2013). With digital textbooks becoming more common across all subject areas (Schuetze 2011), 
there are many opportunities for new and innovative designs that better leverage what we know 
about how students learn. Such leveraging is critical because the change in medium from paper 
to digital textbooks alone will not improve student learning (Russell 1999); the underlying design 
of the tool must be based on theories of how people learn to be effective. In this paper we use a 
learning-theory-driven designto develop a better digital textbook, which in turn provides an avenue 
to examine how digital textbooks can improve student learning. 

The vision for this design began with a well-documented need for engineering students to have 
better conceptual understanding in a variety of core engineering subjects (Streveler et al. 2008). 
In the context of problem solving, there is a common distinction between two types of knowledge: 
procedural ski I land conceptual understanding (Greeno, Riley, and Gelman 1984; Piaget 1978; Rittle- 
Johnson, Siegler, and Alibali 2001). Procedural skill is the “ability to execute action sequences to solve 
problems” (Rittle-Johnson et al., 2001, p. 346), whereas in contrast, conceptual understanding is the 
“implicit or explicit understanding of the principles that govern a domain and of the interrelations 
between units of knowledge in a domain” (Rittle-Johnson et al., 2001, p. 346). While both procedural 
and conceptual knowledge can be used to help solve problems, only those who possess conceptual 
understanding can express ‘why’ and ‘how’ decisions are made in the problem solving process (Piaget 
1978). Students with conceptual understanding have deeper knowledge of the domain, leading to 
more effective problem solving, particularly with unfamiliar problem types (Davenport et al. 2008; 
Karagoz and Cakir 2011; Leppavirta, Kettunen, and Sihvola 2011; Rittle-Johnson, Siegler, and Alibali 
2001). Because of this, conceptual understanding supports innovation and is critical to the success 
of engineering students and practitioners (Rittle-Johnson, Siegler, and Alibali 2001). 

Despite the importance of conceptual understanding, research has repeatedly shown that students 
have misconceptions in a variety of core engineering subjects even after they take courses on those 
subjects (Clement 1982; Halloun and Hestenes 1985; Montfort, Brown, and Pollock 2009; Olds et al. 
2004; Reif and Allen 1992; Scott, Peter, and Harlow 2012). One method that has been shown to pro¬ 
mote the formation of conceptual understanding within the context of digital content collections 
is the use of expert-generated concept maps as advance organizers (Nesbit and Adesope 2006). 
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Figure 1. A concept map of the content in the engineering structures chapter of an 
engineering statics textbook. 


Expert generated concept maps are believed to help students form conceptual understanding by 
giving cues to the learner on how to meaningfully connect the new information to existing prior 
knowledge. An example of an expert-generated concept map is shown in Figure 1. Expert-generated 
concept maps can easily be integrated into these digital content collections as navigation pages that 
link to more detailed content pages, but the large size of the maps create scalability issues. Viewing 
smaller concept maps (less than 20-30 nodes) has been shown to promote conceptual understand¬ 
ing in students, but such maps are often insufficient to represent the content of a course textbook. 
Conversely, students encounter what is called “map-shock” when viewing the kind of larger concept 
maps that might more fully capture textbook knowledge structures (Blankenship and Dansereau 
2000). Map-shock is a type of cognitive overload that prevents students from effectively processing 
the concept map, thereby inhibiting their ability to learn from it. 

One key to increasing the scalability of expert-generated concept maps, then, is to find a way to 
manage cognitive load and there by reduce or eliminate map shock. This idea formed the basis for 
the design of our project. The methods used in this project to manage cognitive load were taken 
from information visualization, a field centeredon using creative visualization schemes to help users 
gain insight from large data sets (Card, Mackinlay, and Shneiderman 1999). 

Based on the need to increase conceptual understanding in engineering students while avoid¬ 
ing map shock, the Adaptive Map tool was designed to help students develop such understanding 
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through an innovative tool for digital content repositories that used information visualization tech¬ 
niques to create a usable, large-scale concept-map-based navigation system for digital content 
repositories. The overarching goal for this research was to address the question: 

How can large-scale concept map visualization tools be realized in a way that promotes concep¬ 
tual understanding and manages cognitive load? 

To refine this open ended question, a prototype tool was created and used by students in an 
engineering statics class to answer four research questions: 

7. How are students engaging in learning activities with the proposed toot? How does this differ 
from the learning behaviors students engage in with traditional textbooks? 

2. How do these observed learning behaviors promote or hinder the formation of conceptual 
un derstanding ? 

3. Are students experiencing cognitive overload during any of the observed learning behaviors, 
and if so how is this affecting the learning process? 

4. What factors contribute to students adopting or not adopting the proposed tool for learning 
activities? 


THEORETICAL FRAMEWORK 

Figure 2 shows a concept map outlining the theoretical framework behind this research and in 
doing so also serves as illustration of an expert-generated concept map being used as an advance 
organizer. 

Promoting Conceptual Understanding through Meaningful Learning 

As discussed earlier, the goal of the Adaptive Map tool is to promote conceptual understand¬ 
ing in students. The tool seeks to help students develop conceptual understanding via meaningful 
reception learning. Meaningful learning relates to conceptual understanding in that students must 
engage in meaningful learning activities to form conceptual understanding (Novak 2002). According 
to Ausubel (1963; 1968), two key characteristics of learning experiences affect how newly learned 
information is stored, retrieved, and used. The first and most influential characteristicis how learn¬ 
ers process the new information. According to Ausubel, the type of processing for any learning 
experience lies on a spectrum between pure rote and pure meaningful learning. In rote learning, 
the learner only minimally processes the information, resulting in poorly connected knowledge, 
whereas in meaningful learning, the information is deeply processed, which results in highly intercon¬ 
nected knowledge structures. Returning to conceptual understanding, meaningful learning leads to 
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Figure 2. The Theoretical Framework for the Adaptive Map Project. 


procedural skill with conceptual understanding, while rote learning leads to procedural skill without 
conceptual understanding. 

The second characteristic addresses how information is presented to the learner, which lies on a 
spectrum between pure discovery and pure reception learning (Ausubel 1963; Ausubel 1968). Pure 
discovery learning results from active experimentation with no expert guidance. Learners form and 
test hypotheses based solely on their own experiences. Pure reception learning occurs when an 
expert presents information to the learner in more or less final form, with no experimentation on 
the part of the learner. Most school learning fall somewhere in the middle, with some expert guid¬ 
ance and some active experimentation, though more traditional programs tend to favor reception 
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Figure 3. Ausubet’s Learning Experience Characteristic Axes. Based on a Diagram in 
(Novak 2010). 


learning. Ausubel states that although these two factors in learning can affect one another, they are 
in fact separate dimensions (Ausubel 1968) that can be represented as axes on a graph, with differ¬ 
ent activities falling into different quadrants (Figure 3). As the figure shows, it is possible to have 
meaningful and rote processing for both reception and discovery learning experiences, and vice versa. 

Ausubel’s work is particularly relevant to the design of a content repository because the repository 
is a representation of domain experts’ knowledge that functions as a reception learning tool - i.e., it 
presents expert knowledge in more or less final form. Unlike most studies on developing conceptual 
understanding, which use discovery or guided discovery learning, Ausubel focused on meaningful 
reception learning and sought to format texts to promote such learning (1968). He developed the 
advance organizer specifically to promote meaningful reception learning from texts. 

Concept Maps as Advance Organizers for Meaningful Learning 

Ausubel’s advance organizers were designed to help students connect what they were learning 
to what they already knew (Ausubel, 1968). They fulfilled that function by giving the learner an over¬ 
view of information being presented at the beginning of the reading to help build a basic cognitive 
framework-a skeleton for the information to be learned. By helping the learner to move from broad, 
general- but correct- cognitive frameworks to more detailed ones, the advance organizer pushed 
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students to engage in meaningful learning. The learner would add to and clarify their initial cognitive 
framework, supplied by the organizer, rather than building a detailed framework from scratch as they 
read. To test these effects, Ausubel and his colleagues conducted numerous studies with university 
students and found that the advance organizers improved both meaningful learning and the reten¬ 
tion of the information (Ausubel 1960; Ausubel and Fitzgerald 1961; Ausubel and Fitzgerald 1962; 
Ausubel and Youssef 1963). A later meta-analysis of 152 studies on advance organizers “. . . found 
a small, but facilitative effect on learning and retention” (Luiten, Ames, and Ackerson 1980, 217). 

The most prominent tool available for creating graphical advance organizers is the concept 
map. Concept maps were first developed in 1972 by Novak and his colleagues as a way to visualize 
what students did and did not know (Novak and Canas 2008). Concept maps are node-link dia¬ 
grams designed to visualize the cognitive schemata of either experts or students (see Figure 2). 
The nodes represent concepts while the links represent the relationship between the concepts. A 
proposition, which is two concepts connected by a single link, serves as a statement that the viewer 
understands. Over the past two decades, numerous studies have explored concept maps as learn¬ 
ing tools; across a variety of settings, grade levels, and content areas, concept mapping has been 
shown to be an effective learning tool (Nesbit and Adesope 2006). Of particular interest to this 
study, expert-generated concept maps have been shown to act as a particularly powerful type of 
advance organizer (Novak 2010). 

To leverage these outcomes for current learning technologies, expert-generated concept maps can 
easily be integrated into digital content collections as navigation pages, but problems with scalability 
exist. While viewing concept maps of less than 20-30 nodes promotes conceptual understanding, 
students encounter what is called “map-shock” when viewing larger concept maps (Blankenship 
and Dansereau 2000). Map-shock is a type of cognitive overload that results in students not being 
able to process the concept map and therefore not learning from it. This loss of benefits for large- 
scale maps hinders widespread use because the large, complex cognitive schema required for full- 
scale engineering courses cannot be represented via a traditional concept map without creating 
map-shock. The key to increasing the scalability of expert-generated concept maps, then, is to find 
a way to manage cognitive load and eliminate map shock. To do so, this study turns to information 
visualization, a field centered on using creative visualization schemes to help users gain insight from 
large data sets (Card, Mackinlay, and Shneiderman 1999). 

Overcoming Map-Shock through Information Visualization 

Map-shock is a specific incarnation of the broader phenomenon of cognitive overload, which is 
based on the proposition that individuals have a limited amount of working memory, and therefore a 
limited rate at which they can process information (Paas, Renkl, and Sweller 2004; Sweller 1988; Sweller 
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2005). Map-shock results in incomplete processing of the map (Blankenship and Dansereau 2000), 
which matches the main effect of cognitive overload. In addition, the affective reactions associated 
with map-shock (Blankenship and Dansereau 2000) also align with cognitive overload (Kirsh 2000). 

Research shows, however, that cognitive overload can be avoided through information visual¬ 
ization techniques (Card, Mackinlay, and Shneiderman 1999), which aim to provide usable ways 
of navigating large amounts of information and develop insights from that information faster and 
more accurately. A properly designed information visualization tool for a large-scale concept map 
should prevent map-shock by managing cognitive load. Within the information visualization litera¬ 
ture, many studies deal specifically with node-link diagrams such as concept maps (as reviewed in 
Herman, Melanpon, and Marshall 2000). These studies discuss techniques to optimize the layout and 
presentation of node link diagrams for easy comprehension and innovative ways to allow the user 
to interact with the concept maps that allow them to easily and quickly find the details they need. 


PROTOTYPE DEVELOPMENT 

To better understand how large-scale concept map visualization tools can promote conceptual 
understanding and manage cognitive load, we developed and tested a prototype. Prototype develop¬ 
ment began by synthesizing the advance organizer, concept mapping, and information visualization 
literature to identify relevant design guidelines. Both content for the Adaptive Map tool and software 
to visualize the content were then created and are briefly summarized in the following sections. 
Technical details on the content development process (Moore, Pierce, and Williams 2012) and the 
software development process (Moore et al. 2013) are available in previous literature. 

Design Guidelines 

The core design guidelines for the Adaptive Map Tool were as follows: 

1. The tool should act as a digital content repository with a concept map based navigation scheme. 
It must be able to display detailed information as well as concept maps of the embedded infor¬ 
mation at several levels of detail and abstraction, and it must allow the user to easily navigate 
horizontally (from one topic to another) and vertically (from one level of detail to another). 

2. Usability of the visualization as both a learning and navigation tool is paramount. The informa¬ 
tion should be displayed in a way that does not cognitively overload the user. 

3. The tool should automatically generate visualizations based on metadata from the content 
developer. The content developer developed content pages with metadata to identify what 
other topics a given idea is directly related to, what the nature of those relationships are, and 
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what groups the topic belongs to. The software should interpret this information and determine 
how to best visualize the information automatically based on the expert-generated metadata. 

4. The tool should be as modular as possible. Software and content should be kept as separate as 
possible so that future content can be developed without understanding the inner workings 
of the visualization software. This separation should lower the barrier to adoption for content 
developers. 

5. The software should allow the user to adjust settings. Since the ideal parameters for a concept 
map are learner-dependent (Wiegmann et al. 1992), users should be able to change settings 
to match their needs. 

6. The concept map visualizations should provide good “symmetry” and “predictability”, as 
these have been shown to improve map processing (Wiegmann et al. 1992; Herman, Delest, 
and Melangon 1998; S. C. North 1996). The displays should present node-link structures with 
similar relationships in similar ways. The displays should also be consistent temporally - if the 
user leaves a view and then later comes back to it, it should look the same as it did the first 
time. Similarly, small changes to content should not radically alter the output. 

7. The concept map should minimize link lengths and link crossings to provide good proximity 
for directly linked nodes and reduce the mental effort needed to interpret links (Purchase 
1997; Ware 2004; Wallace and West 1998; Wiegmann et al. 1992). 

8. The concept map should provide good continuation. Directional links should consistently go in 
one general direction as this makes concept maps easier to interpret (Wiegmann et al. 1992). 
This direction should be vertical to further improve readability (Dansereau 2005). 

9. The nodes and links should utilize color and shape to indicate different features of the nodes 
and links. Concept maps that have been embellished with colors and shapes have been shown 
to out perform unembellished maps (Wallace and West 1998). Embellishments should be kept 
simple, however, and the degree of color and shape variation should be under user control 
because the ideal degree of embellishment varies by user (Wiegmann et al. 1992). 

10. The tool should help the user maintain a sense of context while viewing details because loss of 
context is a common problem when exploring large data sets (Herman, Delest, and Melangon 
1998; Spence 2007). Proven approaches include smooth zooming and panning (Bederson and 
Hollan 1994), multiple window setups relating details and context (Card, Pirolli, and Mackinlay 
1994), and distorted views (Lamping, Rao, and Pirolli 1995; Rao and Card 1994), either indi¬ 
vidually or in combination. 

11. Upon startup, the tool should display an overview of all available content to encourage 
exploration and improve performance in information seeking tasks (C. North 2005; Hornbaek, 
Bederson, and Plaisant 2002; Shneiderman 2010). 


SUMMER 2015 


9 






ADVANCES IN ENGINEERING EDUCATION 

Effectiveness of Adaptive Concept Maps for Promoting Conceptual Understanding: 

Findings from a Design-Based Case Study of a Learner-Centered Tool 


Content Development Process 

The second element in the prototype design was the development of content: useful informa¬ 
tion visualization is predicated on useful information. The content development process began by 
creating a detailed concept map for an engineering statics course. Since the existing procedure to 
create concept maps (Novak and Canas 2008) was inadequate for the scale of the desired map, a 
new procedure was developed and validated (Moore, Pierce, and Williams 2012). 

Content development also required detailed content associated with each topic in the map. These 
were traditional html pages that included text, images, and worked problems similar to that found 
in traditional textbooks. Multimedia and interactive elements were not included on these pages to 
allow a more useful comparison of the Adaptive Map tool to a traditional textbook. 

Overview of the Adaptive Map Tool 

The final prototype Adaptive Map tool presents material at multiple levels of abstraction, rang¬ 
ing from a concept map presenting broad overview of all subjects (Figure 4A), to concept maps 
for individual content sections (Figure 4B), to detailed text on each topic (Figure 4C). Through this 
“layered” graph algorithm (in the language of Herman, Melanpon, and Marshall 2000), the concept 
map format serves as an advance organizer. The visual presentation of the map is controlled by 
the user’s level of abstraction (how zoomed in or out they are) and the focus topic. Based on this 
user-selected information, the software presents a visual of the most relevant topics to the current 
focus at an appropriate level of detail for the current level of abstraction. 

Users interact with the tool by clicking and dragging around the background to pan, using the 
scroll wheel or the + and - buttons to zoom out, and clicking or double clicking on nodes to center 
on the node and open the node respectively. Through these inputs, the users can easily and intui¬ 
tively choose their focus and level of detail within the large-scale concept map. 

A variety of information visualization techniques help the users quickly and easily process map 
information. 

• The tool starts by presenting users an overview of all topics, helping users develop a feel for 
the overall data set. 

• Color cues indicate different clusters of information, with the background color in mid-level 
views (Figure 4B) indicating the current focus cluster. 

• Links show relationships, with the width of the linking line (Figure 4A) showing the strength 
of the relationship. 

• Semantic zooming techniques are used in the mid-level view (Figure 4B) to distort the concept 
map so that links are shortened and the topics most closely related to the focus topic are 
drawn to the center of the screen. 
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Method of Joints 


The method of joints is a technique used to solve for the forces 
acting on all members of a truss structure. As the name implies, the 
method of joints focuses on the joints where the members of the 
truss structures come together This method can be used to solve 
for the internal forces in both plane and space trusses. 

Method of Joints for Plane Trusses: 

In a plane truss, think about the pins in the pin joints that connect all 
the members We will calculate the equilibrium equations of each of 
these pins in order to calculate the forces on each of the members 
in the truss structure. The process below outlines the steps in the 
method of joints. 


6 . 
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Fe 

•fc 


The method of joints analyzes the pins that 
connect the members in the truss structure. 


1. Make sure that the structure is a statically determinate truss 
structure (as outlined on the truss page) and that all external 
forces act on the joints, not the middle of one of the 
members. If these assumptions are not met the method of 
joints will not produce valid results 

2. Draw a free body diagram of each of the pins in the structure. Include the external load forces, the external 
reaction forces, and the forces that each member exerts on each pin. Remember that each member is a two 
force member so the members exert a force along the line between the two connection points on the member 
You will also need to guess whether the member is in tension or compression. Members in compression will 


Figure 4. The Adaptive Map Toot. 
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• On all view levels, the users can get more detailed information by hovering over the nodes or 
links. This hides information that would be cluttering up the screen, but provides easy access 
to “details on demand” for the users. 

• Finally, to help the student maintain a sense of context while navigating, the tool presents 
smooth transition animations when the user is changing either their focus topic or the level 
of zoom. 

To explorethe finished software with engineering statics content, readers can download the work¬ 
ing tool at adaptivemap.ma.psu.edu , view a video of the software in use at that same site, or down¬ 
load the software itself (which is open source) from https://github.com/JacobMoore/Adaptive-Map . 


METHODS, DATA COLLECTION, AND DATA ANALYSIS 

To test the effectiveness of the tool and answer the research questions, amultiple case study ap¬ 
proach was adopted (Yin 2009). A case was defined as a single student using either the Adaptive 
Map tool or a traditional paper textbook over the course of the semester in engineering statics. The 
inductive case study approach was well suited to understand how students used and learned from 
the Adaptive Map tool in an unstructured out-of-the-classroom context (Creswell 2009). 

Participants 

Fourteen participants volunteered for the study, of which nine participated in all test sessions to 
yield complete data sets. The students were selected from two similar sections of statics. They had 
the same instructor, were offered in the same semester, and each had approximately 150 students. 
The adaptive map tool was introduced to one section at the beginning of the semester while the 
tool was not introduced to the other section. All students in the Adaptive Map section (whether 
they volunteered or not), were able to access the online tool at their convenience outside of the 
study and the classroom. Students in the Adaptive Map section were introduced to and allowed to 
use the tool before being asked to volunteer for the study. All adaptive map volunteers were pulled 
from the section that was introduced to the tool, while all traditional textbook students were pulled 
from the section not introduced to the tool. 

Each participant provided informed consent and was compensated in the form of Amazon gift 
cards for the time they spent. An experienced statics instructor was also asked to complete some 
of the same tasks and served as an additional contrasting case, for a total of ten cases. Each par¬ 
ticipant was assigned a pseudonym to be used in all analysis in order to protect the identity of the 
participants. Six students used the Adaptive Map tool and served as the primary cases, while three 
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students used a traditional paper textbook (Meriam and Kraige 2007) and served as contrasting 
cases. Having both primary and contrasting cases provides a means to compare the experiences of 
the Adaptive Map tool users to the experiences of traditional textbook users and to the experiences 
of an expert in the content area (Yin 2009). 

Data Sources and Procedures 

Table 1 outlines the research questions, the data sources mapped to each question, and the cor¬ 
responding analytical methods. The data sources covered general student data, data on tool usage 
outside the data collection sessions, detailed information on tool usage under controlled circum¬ 
stances, and student learning. 

Data collection primarily followed a process outlined in the Center for Research on Evaluation 
Standards and Student Testing (CRESST) data collection sessions (Baker et al. 1992). These sessions 
began with the researcher asking the participant to study a specific topic area, and then teach that 
topic to the researcher. Two of these sessions were conducted during the semester, the first session 
addressing a topic previously covered in class (engineering structures) and the second addressing a 
topic that was beyond the course lectures (fluid statics). A single researcher conducted all sessions. 
The sessions were conducted individually with each participant. For each session, the participant had 
twenty minutes to prepare their lesson with the Adaptive Map tool (for the Adaptive Map group) or 



Research Question 

Data Source 

Analysis Method 

How are students engaging 
in learning activities with the 
proposed tool? How does this 
differ from the learning behaviors 
students engage in with traditional 
textbooks? 

Weekly usage surveys 

Results compiled into usage summary table. 

Semi-structured 
debriefing interview 

Interviews recorded, transcribed, thematic analysis 
conducted. 

CRESST preparation 
observation notes 

Usage patterns reported. 

How do these observed learning 
behaviors promote or hinder 
the formation of conceptual 
understanding? 

CRESST explanation 
sessions 

Sessions recorded and transcribed with notes. 

Transcripts coded. 

Transcripts scored according to CRESST rubric. 

Concept map of student understanding made from transcript. 

Statics Concept Inventory 

Scores reported. 

Are students experiencing cognitive 
overload during any of the observed 
learning behaviors, and if so how is 
this affecting the learning process? 

NASA TLX survey 
instrument 

Results compiled into cognitive load summary table. 

Descriptive statistics calculated. 

What factors contribute to students 
adopting or not adopting the 
proposed tool for learning activities? 

Semi-structured 
debriefing interview 

Interviews recorded, transcribed, thematic analysis 
conducted. 

Table 1. Research Questions, Data Sources, and Analysis Methods. 


SUMMER 2015 


13 


















ADVANCES IN ENGINEERING EDUCATION 

Effectiveness of Adaptive Concept Maps for Promoting Conceptual Understanding: 

Findings from a Design-Based Case Study of a Learner-Centered Tool 


the traditional textbook (for the traditional textbook group). Care was taken to pick sections of the 
Adaptive Map tool and traditional textbook that covered identical content areas. All concepts and 
procedures were discussed in similar ways (matching the adaptive map content to the textbook con¬ 
tent) and the adaptive map did not include any videos or interactive elements other than the concept 
map navigation system. Care was taken to minimize differences between the content in each tool. 

During this preparation time, the researcher observed the participant and was seated such that 
they were able to see what parts of the Adaptive Map tool or textbook the student was looking at. 
The researcher took careful notes on what sections the student was looking at, how long they spent 
looking at each section, when the student took notes, and any other student behaviors that stood out. 

After the preparation session, the participant completed the NASA Task Load Index (Hart and 
Staveland 1988), which measures a number of workload characteristics, including cognitive load. 
The instrument has been extensively used and validated in the literature (Hart and Staveland 1988; 
Hart 2006) and provided a simple measure of the cognitive load imposed by the tool (Adaptive 
Map or textbook). 

The researcher conducting the sessions acted as the target ‘learner’ and participants taught the 
researcher about the selected topic. The researcher acted as a novice, letting the participant lead 
the discussion and asking probing questions as needed when the explanations were unclear. In do¬ 
ing so, the researcher used clinical interviewing techniques (Ginsburg 1997) to allow for probing 
questions without leading or biasing the participant. The explanation session was audio recorded 
and transcribed, and diagrams drawn during the explanation were scanned and included at appro¬ 
priate points in the transcript. 

Other data sources included short weekly surveys that tracked how often and for what purposes 
participants used the Adaptive Map tool or traditional textbook, a debriefing interview that more 
deeply explored usage patterns outside of the observed research sessions and reasons participants 
chose to adopt or not adopt the tool, and the Statics Concept Inventory (Steif and Dantzler 2005), 
an instrument designed and validated to broadly measure conceptual understanding of engineering 
statics content. The survey was administered every week over the course of the study, and both the 
debriefing interview and the Statics Concept Inventory were administered at the end of the semester 
to capture the effects and opinions of the tool over the course of the entire semester. 

Analysis 

The CRESST explanations were analyzed using two separate techniques. First the transcripts were 
evaluated according to a modified CRESST rubric on a five point scale addressing content coverage, 
relevant prior knowledge, conceptual understanding, and procedural knowledge. This rubric provided 
high-level measurements of various aspects of the student’s knowledge related to the topic at hand. 
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The second method involved coding the transcripts, then creating the concept maps of each 
student’s explanations. This approach provided a much richer picture of what each student did 
and did not understand that could be linked to specific aspects of the student’s preparation. Each 
transcript was first coded for the type (conceptual, procedural, etc.) and the correctness of the 
explanation (correct, incorrect, avoided questions, etc.). These coded transcripts were then used 
to create a concept map of each student’s explanation. The process was modeled on the work of 
Novak and Musonda (1991) in converting clinical interview transcripts into concept maps. Each ma¬ 
jor topic discussed was marked as a node, while each discussion of a direct relationship between 
two topics was marked as a link. After the concept map was built, colors were used to encode the 
correctness of the explanations. Examples of completed explanation concept maps can be seen in 
Figures 6-8 under Results. 

All transcripts were coded by a single researcher and the same researcher created all explanation 
concept maps. A code-recode reliability check (Carey, Morgan, and Oxtoby 1996) was conducted for 
the coding procedure and a map-remap reliability check (Novak and Musonda 1991) was conducted 
on four of the eighteen total transcripts. Both of these checks were conducted by the original coder 
and were conducted at least one month after the original coding and mapping. The researcher 
achieved an agreement level of .90 for the code recode check and an agreement level of .90 for the 
map-remap check. Both of these measurements indicate the researcher demonstrated a high level 
of reliability in coding and mapping the data. 

Finally, within-case and cross-case analyses were conducted based on the analyses of all data 
sources for each participant. First individual case profiles were created, representing a narrative for 
each participant that synthesized all the data collected for that participant. To ensure the complete¬ 
ness of each profile and to allow for easier contrasting of the profiles in later analysis, a case profile 
format guide was used (Yin 2009). After all the case profiles were created, the researchers looked 
for common themes or patterns across the case profiles. 

Through this multi-step process, the researchers distilled the rich set of raw data into central 
themes that illuminate how the Adaptive Map tool was used, how it affected student learning, and 
how the tool could be improved. 


RESULTS AND DISCUSSION 


Two primary themes emerged in the data relative to the CRESST sessions: “advance organizer 
usage” and “prior content exploration.” Advance organizer usage refers to the degree to which 
students were observed to view and process an overview of the information before turning to more 
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Figure 5. Student Advance Organizer Usage and Prior Content Exploration. 


detailed content information, while prior content exploration refers to the degree to which students 
viewed and processed information outside of the assigned content area (usually prerequisite top¬ 
ics to the section). The two issues can be treated as orthogonal. Each participant was rated on a 
four point scale for each topic, and Figure 5 shows the resultant quadrant diagram. Underlined 
participants used the Adaptive Map tool; italicized participants used the traditional textbook. As 
the figure shows, students using the Adaptive Map tool were more likely to exhibit behaviors that 
indicate advance organizer usage and prior content exploration. 

Advance Organizer Usage 

The first theme centers on if and how students used the concept maps in the Adaptive Map tool 
or the section headings in the traditional textbook as an advance organizer. The higher the score on 
the advance organizer usage scale, the more time the participant spent viewing and interacting with 
an overview of the content (observed through direct observation of the CRESST preparation as a 
percentage of the total twenty minute preparation period). That time suggests that they processed 
the overview before processing content details, the key goal of the advance organizer. 

Anne was the clearest example of a student using the features of either tool as an advance or¬ 
ganizer. At the beginning of both CRESST sessions, Anne spent several minutes looking over the 
concept map of the cluster, hovering over the nodes and links to get details, and taking notes while 
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examining the concept map. These behaviors indicate that she spent time processing the overview 
of the content (in the form of a concept map) before moving to the more detailed topic pages. Anne 
began taking notes by spreading a few headings across the page to represent the main ideas, then 
adding details under the headings as she read topic pages. This highest level of advance organizer 
usage was classified as level 1. 

On the other end of the spectrum, Aaron and Carol exemplify students not using an advance 
organizer (classified as level 4). These students read the information in the traditional textbook 
carefully from beginning to end. As there was no overview section present in the traditional text¬ 
book, they did not process an overview of the information in the assigned section at any point. 
Aaron and Carol both took detailed notes on each topic as it was addressed in the textbook, but 
didn’t exhibit any behaviors that would indicate they looked for or thought about an overview of 
the content. 

All other participants fell somewhere between these two extremes. Lillian, Lisa, and Ryan were 
classified as moderate advance organizer users (level 2), spending a minute or two looking through 
an overview of the information (though far less time than Anne). Sonya, Gabriel, and Paul were clas¬ 
sified as light advance organizer users (level 3), viewing and taking notes on part of the overview 
but not all of it. Ryan was of particular interest because he was the only advance organizer user in 
the traditional textbook group. Ryan began each CRESST session by flipping through the section, 
presumably using the section headings to get an overview of the information to come. He then used 
these section headings in the textbook to create section headings in his notes, much the way Anne 
did from the concept map overview. 

Theoretically, advance organizer usage should help students mimic an expert’s cognitive frame¬ 
work for the information (Ausubel 1968; Luiten, Ames, and Ackerson 1980). The organizer should 
make students more likely to meaningfully relate topics to one another as the expert has done and 
less likely to rely on rote learning strategies. The results should be reflected in deeper conceptual 
understanding and more connected cognitive frameworks. To test this hypothesis, participants’ 
organizer usage were compared to their performance on the CRESST rubrics; for this comparison, 
participants were classified as either high (levels 1 and 2 including Anne, Lillian, Lisa, and Ryan) or 
low (levels 3 and 4 including Aaron, Carol, Sonya, Gabriel, and Paul). Table 2 presents the distribu¬ 
tion of CRESST rubric scores for each of the two groups. 

In addition to the full distribution of scores being shown for each group in each category, a Mann- 
Whitney U test (also sometimes called a Wilcoxon Rank sum test) was used to compare the two 
groups. Because of the small sample size and non-normal distribution of scores, the more common 
Student’s T-test was inappropriate, but the Mann-Whitney U test examines the rank of each data 
point in the data set and does not require a normally distributed data set. 
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CRESST 

Category 

Advance Organizer 
Usage Group 

Scores 

Mann Whitney 
Test P-Value* 

Principle 

Low Usage 



.757 

Coverage 

High Usage 



Prior 

Low Usage 



.246 

Content 

High Usage 



Conceptual 

Low Usage 



.046 ** 

Knowledge 

High Usage 



Procedural 

Low Usage 


.897 

Knowledge 

High Usage 




Category Descriptions 

Principle Measure of how complete student was in at least addressing all the 
Coverage topics in the assigned section. 

Prior General measure of the student’s understanding of topics outside of 

Content the assigned section but necessary as a base for the assigned topics. 

Conceptual Measure of the student’s understanding of the concepts within the 
Knowledge assigned section and the relationships between those concepts. 

Procedural Measure of the student’s ability to apply problem solving 
Knowledge methodologies presented in the assigned section. 

*Two-Tailed Mann Whitney U Test 
^^significant at the p<.05 significance level 

Table 2. Comparison of High and Low Advance Organizer User CRESST Scores (Scores 

out of Five). 


Scores 



The biggest difference that is visible is the distribution of the scores for conceptual understanding. 
The “Low Advance Organizer Usage” group was much more likely to have low scores for conceptual 
understanding while the “High Advance Organizer Usage” group was much more likely to get high 
scores for conceptual understanding. This is reflected in the statistically significant result from the 
Mann-Whitney test. In other areas, advance organizer usage did not appear to have an effect. 

Comparing the explanations of a very high advance organizer user in the Adaptive Map group 
(Anne) and a very low advance organizer user in the traditional textbook group (Carol) further il¬ 
lustrates the stark differences between groups. Beyond differences in advance organizer usage, 
these two students shared characteristics that made them good candidates for comparison. Both 
were high performers in the class; Anne had the highest expected course grade “A” and Carol had 
the second highest “A or A minus”. Both were also observed to spend a vast majority of the prepa¬ 
ration time within the prescribed topic area, leading to similar low prior content exploration scores 
and limiting the degree to which prior content exploration differences affected their explanations. 
Scores on the Statics Concept Inventory did differ by a large amount; with Anne having the highest 
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Adaptive Map Traditional Textbook 

Concept Map Concept Map 



Figure 6. Comparison of Anne and Carol's CRESST Session 2 Explanation Concept Maps. 


score (74.1%) and Carol having the fourth highest score (40.7%). Though this represents a significant 
difference in scores, both were also above average for the study volunteers. 

Figure 6 compares the concept maps of Anne and Carol’s explanations from the second CRESST 
session where the differences in explanations were more pronounced, likely because the session 
covered new material and thus depended on learning only within the research session. Concept maps 
of the source material (either the Adaptive Map tool or the traditional textbook) are shown above 
the students’ explanation maps. The maps of the source material were made in the same manner as 
the explanation concept maps, treating the actual content as a transcript. 

As seen in Figure 6, stark differences emerged in the explanations the two students offered. 
Anne’s explanation had no errors, as indicated by the green nodes. Her discussion related key 
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topics to one another, as indicated by the plentiful links in the concept map. Finally her map per¬ 
fectly matched the map of the source content, indicating that Anne understood the material as it 
was covered in the Adaptive Map tool. On the other hand, Carol’s explanation contained a large 
number of errors, as indicated by the red nodes. She rarely discussed the relationships between 
topics, and when she did these relationships were incorrect as indicated by the few red links. 
The maps suggest that Anne seemed to have an understanding of the material in her discussion, 
while Carol seemed to be “throwing around equations” without a real understanding of why they 
worked or even what many of the variables represented. In other words, Carol lacked a conceptual 
understanding of the topic. 

The differences in the explanations of Aaron and Ryan, two textbook users on different ends of 
the advance organizer usage scale, reinforce the contrast. Figure 7 shows the CRESST session 2 
explanation concept maps for Aaron (non-advance organizer user) and Ryan (moderate advance 
organizer user). Aaron, much like Carol, simply started reading and taking notes on the detailed 
content as he read through the information linearly. Ryan, on the other hand, skimmed through the 
section and used the section headings in the textbook to outline his notes, which he then filled in 
by reading the detailed content information later. As Figure 7 demonstrates, Aaron’s explanation 
for session 2 had many errors and was poorly connected, while Ryan’s map shows fewer and more 
minor errors, with more relationships among topics. 

The more connected knowledge seen in Ryan and Anne’s explanation is a sign of conceptual 
understanding (Ambrose 2010; Glaser 1984; Mathematics Learning Study Committee; National 
Research Council 2001; McCormick 1997; Nickerson 1985; Prawat 1989). The cross-case analysis 
suggests, moreover, that students who spend more time viewing and processing an overview of 
the information have more connected explanation concept maps and therefore deeper conceptual 
understanding of the content material studied. These results match with the design intentions of 
the tool to promote conceptual understanding through advance organizer usage. 

To explore factors that may have supported or facilitated advanced organizer usage, data analy¬ 
sis explored relationships between advance organizer usage and i) the frequency of tool use, ii) 
primary tasks performed while using the tool, iii) mastery versus performance learning orientation, 
and iv) overall course performance. 

Only one factor had a clear and strong relationship to advance organizer adoption: the primary 
task performed while using the tool outside of the CRESST sessions. In the Adaptive Map group, 
the three heaviest users of the concept maps as advance organizers (Anne, Lillian, and Lisa) were 
also the three participants who reported, via the weekly usage surveys, using the Adaptive Map tool 
primarily to review information outside of class and to study for exams. This pattern was absent for 
those users who used the concept maps as advance organizers the least (Gabriel, Paul, and Sonya). 


20 


SUMMER 2015 





ADVANCES IN ENGINEERING EDUCATION 

Effectiveness of Adaptive Concept Maps for Promoting Conceptual Understanding: 
Findings from a Design-Based Case Study of a Learner-Centered Tool 




Gabriel and Paul used the tool primarily to look up information to solve homework problems, and 
Sonya did not use the tool at all outside of the research sessions. 

Though it is impossible to assign causality based on the gathered data, these results corroborate 
the findings of Lee et al. (2013) which found that students used their textbooks only superficially 
during problem solving activities. These students engaged a targeted search strategy, relying on 
features such as the search bar, which both Gabriel and Paul mentioned as a positive tool feature. 
By using such features, these participants largely ignored the concept map and jumped directly to 
the details they needed. Those who used the tool to study, however, may have engaged in a more 
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open ended search. Without a clear final destination (solve this problem), they may have given 
more attention to the context provided by the concept map as they moved through the material. 
With a preference for, and more practice in, this type of navigation, students who used the Adaptive 
Map tool to study regularly gave more attention to the concept maps during the CRESST sessions. 
Further investigation is needed to pursue this pattern. 

Prior Content Exploration 

The second major theme addressed content exploration: students using the Adaptive Map tool 
were more likely to explore content outside the assigned section (observed through direct observa¬ 
tion of the CRESST preparation as a percentage of the total twenty minute preparation period), and 
those students who spent more time exploring outside content learned in a way that more closely 
matched expert learning. As described in the methods section, all student participants were given a 
clearly defined section of content to explain for each CRESST session. All traditional textbook users 
read only the information in the prescribed section; they never looked at pages in other chapters or 
even other sections in the same chapter. All Adaptive Map users, on the other hand, viewedat least 
one page outside the prescribed topic area in at least one of the CRESST sessions. These pages 
usually addressed topics that served as prerequisites to the prescribed topic, hence the name “prior 
content exploration.” Some Adaptive Map users, such as Lillian and Sonya, spent a third or more of 
their preparation time reading pages outside the prescribed topic area. Moreover, when questioned, 
these participants clearly knew they were outside the prescribed topic area, indicating intentional 
exploration of outside content not undertaken by the traditional textbook students. 

To more fully explore this phenomenon, students were classified in terms of how much attention they 
gave to outside content. Sonya and Lillian, who spent a third or more of their time on outside topics, 
were classified at the highest level (1). Participants who carefully read through at least one outside 
topic page were place at the next level (2), while those who skimmed at least one outside page were 
at the third level (3). Finally, those who never looked at pages outside the prescribed topic area such 
as Aaron, Carol, and Ryan were at the lowest level (4). These classifications can be seen in Figure 5. 

Table 3 compares the distribution of CRESST rubric scores (across both sessions) for the high 
prior content explorers (Levels 1 and 2: Gabriel, Lillian, Lisa, and Sonya), and the low prior content 
explorers (Levels 3 and 4: Anne, Paul, Aaron, Carol, and Ryan). 

The largest observed difference is in the distribution of principle coverage, with the “Explorer” 
group receiving lower scores in general on the measures of principle coverage while the “Non- 
Explorer” group received higher measures of principle coverage. Though these differences were not 
significant according to the Mann-Whitney U test, these lower principle coverage scores match the 
observed behaviors during preparation. Students who spent a lot of time looking at content outside 
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CRESST 

Category 

Content Explorer 
Group 

Scores 

Mann Whitney 
Test P-Value* 

Principle 

Non-Explorer 



.101 

Coverage 

Explorer 



Prior 

Non-Explorer 



.857 

Content 

Explorer 



Conceptual 

Non-Explorer 



.197 

Knowledge 

Explorer 



Procedural 

Non-Explorer 


.562 

Knowledge 

Explorer 




Category Descriptions 


Principle Measure of how complete student was in at least addressing all the 
Coverage topics in the assigned section. 


Prior General measure of the student’s understanding of topics outside of 

Content the assigned section but necessary as a base for the assigned topics. 


Conceptual Measure of the student’s understanding of the concepts within the 
Knowledge assigned section and the relationships between those concepts. 


Procedural Measure of the student’s ability to apply problem solving 
Knowledge methodologies presented in the assigned section. 


Scores 



*Two-Tailed Mann Whitney U Test 
^^significant at the p<.05 significance level 


Table 3. Comparison of Content Explorers and Non Content Explorers (Scores out of 


Five). 


the assigned section (such as Sonya and Lillian) had less time to read information in the assigned 
section. Those topics that were not covered during preparation were then often not covered in the 
explanations. At the same time, the higher conceptual understanding scores sometimes observed 
in this group (also not statistically significant) mean that for the material covered in the explana¬ 
tions, students in the high content explorer group had better understanding of the information they 
covered. That is, while they did not cover as much material (principle coverage), they understood 
the material they covered more thoroughly. 

Figure 8 illustrates the differences in the explanation concept maps observed between high and 
low content explorers, exemplified by Sonya and Paul respectively. The expert map (James) is also 
included as a reference point. As in past comparisons, Paul and Sonya are comparable in many 
respects. Both used the Adaptive Map tool and therefore had the same source content, both were 
average in terms of overall academic achievement, and neither was at either extreme of observed 
advance organizer usage. 

In their explanations, as Figure 8 shows, both Paul and Sonya have some minor errors and both 
are missing some links (Paul) or topics (Sonya), but the structures of their maps differ. Sonya has a 
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Paul's Explanation Concept Map (Adaptive Map, 
Non-Explorer) 



Map, Explorer) 



James's Explanation Concept Map 
(Textbook, Expert) 


Key 

No Errors Discussing Topic 
Minor Errors Discussing Topic 
Major Errors Discussing Topic 

Incomplete Coverage of Topic 

Dashed boxes represent the extent of the assigned content 


Figure 8. CRESST Session 2 Prior Content Explanation Comparison. 
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firm grasp of the more basic topics (top of the map) but lacked understanding in the more advanced 
topics (bottom of the map). On the other hand, errors and omissions seemed to be distributed more 
evenly throughout Paul’s explanation. This difference seems to indicate a difference in strategies 
by which Sonya mastered lower-level topics before moving on to more advanced ones, even if that 
meant not getting to the most advanced topics, while Paul seemed to concentrate on getting through 
all the material during the preparation session even if he misunderstood a few topics along the way. 

Notably, Sonya’s map more closely resembles that of James. James was an experienced statics 
instructor, but, like the students, did not cover fluid statics in his class and therefore also had to learn 
that material for the second CRESST session. James used the traditional textbook to prepare, unlike 
the two students, but like Sonya, James seemed to have a solid foundational understanding of the 
basic information, with mistakes clustered in the more advanced topics. Explanation maps for both 
of the highest content explorers (Sonya and Lillian) mirrored the expert’s map pattern (strong base 
knowledge, errors in advanced topics) and thus likely also mirrored the expert’s learning strategy. 

In exploring factors that might explain these prior content exploration patterns, five factors were 
considered: i) the tool itself, ii) frequency of tool use, iii) primary tasks performed while using the tool, 
iv) mastery versus performance learning orientation, and v) overall course performance. Of these, 
two emerged as salient. The most obvious and influential factor was the tool: students using the 
Adaptive Map tool engaged in prior content exploration while those using the traditional textbook 
did not. This difference suggests that the Adaptive Map tool supports prior content exploration for 
novice learners in a way the traditional textbook does not. The second salient factor was the Statics 
Concept Inventory score. Anne and Paul had the highest scores, but spent the least time exploring 
prior content. The high explorers (Gabriel, Lillian, Lisa and Sonya) scored below the average for 
all research participants. Though potentially confounding, the lower scores may indicate a higher 
need for exploration. That is, students coming into the CRESST sessions with prior understanding 
of the course content (such as Anne and Paul) did not need to explore previously covered material. 
In contrast, students who did not understand the prior content as well before the sessions were 
more likely to look at topic pages supporting the prescribed topic area. This relationship suggests 
that the Adaptive Map Tool may be particularly useful for helping students who are struggling with 
course concepts, though further research would be required to test that hypothesis. 


DESIGN AND IMPLEMENTATION ISSUES 

While the two dominant themes (advance organizer usage and prior content exploration) address 
student learning patterns with respect to the Adaptive Map Tool, three additional themes emerged 
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from the data that shed light on the design and implementation of the tool: cognitive load, content 
language complexity, and tool adoption. 

First, and most importantly, there was no observed difference in cognitive load between the 
tool and the textbook. That is, using the 90+ node statics concept map with the Adaptive Map tool 
was no more mentally demanding than using a traditional textbook, indicating that the information 
visualization techniques designed to help users manage cognitive load were successful. Notably, 
students who reported particularly high levels of cognitive load (regardless of tool) on the NASA 
TLX instruments (Aaron, Gabriel and Paul) did poorly on the corresponding CRESST explanation 
activities. This correlation further supports the notion that high cognitive load can in fact interfere 
with learning and should be managed appropriately. 

Second, many students identified the simpler and more conversational language of the Adaptive 
Map tool (in contrast to the more formal language in the traditional textbook) as a draw. In fact, in the 
debriefing interview, the tone of the language was one of the two most commonly identified benefits, 
(map layout being the other). The two participants that self-identified as non-native English speakers 
(Anne and Gabriel) found the language particularly beneficial. Notably, this difference in tone was 
not intentional; as such it serves as both a potentially confounding factor and an area for future work. 

Third, with respect to implementation, adoption rates were high within the participant pool, 
where five out of six participants reported using the tool outside the research sessions once a week 
or more. Website analytics confirmed their usage, but also showed few users outside the research 
participant pool, even though the tool was introduced to about 150 students outside the pool. One 
explanation may be that study participants adopted the tool after they became familiar with the 
interface during the research sessions; for non-participants, the cost required to learn something 
unfamiliar coupled with the lack of structured exposure may have limited students’ willingness to 
explore and adopt the tool. This finding suggests that while the tool design does provide a usable 
interface that can support learning, it may require structured engagement to encourage use, par¬ 
ticularly if students have an alternative, more familiar tool at hand. 

DISCUSSION AND CONCLUSIONS: DESIGNING DIGITAL LEARNING MATERIALS 

As noted in the Introduction, this study sought to identify ways to realize concept map visual¬ 
ization tools in ways that promote conceptual understanding while managing cognitive load. The 

four research questions that framed the study work together to address this goal. 

RQ1: How are students engaging in learning activities with the proposed tool? How does 

this differ from the learning behaviors students engage in with traditional textbooks? 
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Students using the Adaptive Map tool are more likely to exhibit behaviors that indicate advance 
organizer usage and prior content exploration than students using a traditional textbook. That is, 
they are 1) viewing and processing an overview of the information before trying to process the de¬ 
tailed content information, and 2) reviewing topics from previous sections that are relevant to the 
section they are currently studying. 

RQ2: How do these observed learning behaviors promote or hinder the formation of 

conceptual understanding? 

Both advance organizer usage and prior content exploration were linked to higher measures 
of conceptual understanding. By supporting these practices, the Adaptive Map tool appears to 
encourage students to engage in meaningful learning that leads to the formation of conceptual 
understanding. 

RQ3: Are students experiencing cognitive overload during any of the observed learning 

behaviors, and if so how is this affecting the learning process? 

Students using the Adaptive Map tool did not seem to experience higher cognitive loads than 
students using the traditional textbook. Students who reported particularly high cognitive loads 
in either group, though, performed poorly on learning measures, reaffirming the value of cognitive 
load theory in this context. 

RQ4: What factors contribute to students adopting or not adopting the proposed tool for 

learning activities? 

Students identified the tool’s map layout, simple and more conversational language, and search 
feature as primary reasons for adoption. Notably, research participants adopted the tool at a high 
rate, but few outside this group did so, despite being provided with access. Increasing adoption 
rates likely requires improving the interface to increase usability and/or integrating the tool more 
tightly into a class to increase students’ familiarity with the interface. 

Together, these findings address the primary goal: An Adaptive Map tool designed using infor¬ 
mation visualization techniques helps users manage cognitive load while viewing large concept 
maps. In doing so, such a tool can support student use of concept maps as advance organizers and 
encourage students to explore relevant previously covered content. Both behaviors support the 
meaningful learning that leads to the formation of conceptual understanding. 
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These results suggest that, as content becomes increasingly digital, adaptive maps such as the 
one tested in this study can play a critical role in helping students navigate and learn from these 
digital repositories. At the same time, the findings go beyond the Adaptive Map tool itself and sug¬ 
gest implications for the design of digital learning aids more broadly. 

• Deliberately design and present an overview of the information : The Adaptive Map was de¬ 
signed primarily as an advance organizer, and the designers deliberately addressed decisions 
needed to visually present an understandable overview of the information in the content 
repository. The students who were most successful in learning the information, regardless of 
tool, were the ones who utilized this overview. Designs should encourage students to view 
and process a meaningful visual overview of information, targeted toward novices, before 
addressing detailed content. 

• Make relevant prior information easily accessible : All students using the Adaptive Map tool 
viewed relevant information covered earlier in the course, while students using the traditional 
textbook did not. This act of looking back enhanced the learning process. The visual links in 
the Adaptive Map tool made it very easy to visit related prior information quickly and then 
return to the topic at hand; traditional textbooks, in contrast, require both knowing what those 
links are and knowing where to find them in previous chapters. By making it easy for users to 
locate and visit relevant topics and then quickly return to their original location, digital tools 
can help students connect the new topics to the relevant prior content. 

• Help users manage cognitive load : A central design goal of the Adaptive Map tool was to help 
users manage their cognitive load when examining large stores of information. The tool was 
largely successful, but students who did report high cognitive loads using either the Adaptive 
Map tool or the traditional textbook had lower measures of learning across all categories. As 
a result, designers must continue to manage visual information, as suggested by the Design 
Guidelines, avoiding screen clutter that could overwhelm the users and thus hinder learning. 

• Identify the sources of usage behaviors and habits : Not all students who used the Adaptive 
Map tool used it in the same way, and their research session usage appeared linked to the 
way they used the tool outside of the research session. These outside usage patterns seemed 
to be developed into habits that were carried into the research sessions. As a result, in stud¬ 
ies of digital learning tools, researchers must look beyond the behaviors students exhibit in 
controlled settings to explore how and why students developed those behaviors. 

• Make novel design features intuitive and give students time to adjust : As noted, although the 
Adaptive Map tool was introduced to over 150 students, only the study participants showed 
meaningful adoption; notably, adoption seemed to result from participation in the study, rather 
than the reverse. In other words, students needed time and support to become familiar with 
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the tool - here provided by the CRESST sessions - in order to adopt it. Thus two avenues to 
support adoption of novel digital tools would be to 1) tightly integrate the tool through in-class 
usage or homework problems to give students time to learn the interface, and/or 2) make the 
interface design as intuitive as possible to minimize the learning curve. 

These guidelines, emerging from the study of students’ engagement with one particular digital 
tool, provide an important framework as more and more educators develop digital learning tools 
for their courses. 


LIMITATIONS AND FUTURE WORK 

This study fills a number of gaps in the literature regarding the application of information visual¬ 
ization, advance organizers, concept mapping, and cognitive load theory to student learning, but it 
also raises many new questions. First, this work was limited in scope; it looked at a small sample of 
students in a single course with a single instructor. This specially selected population helped limit 
the number of factors that had to be considered during analysis and allowed for a deep exploration 
of how participants used the tool, but further work across more participants is needed to examine 
how the Adaptive Map or similar learning tools are used by different students, in different classes, 
with different instructors. Additionally, this study compared a single digital tool to a single tradi¬ 
tional textbook, and the specificcontent and the layout of these two tools are invariably tied to the 
results of this study. Consequently, further studies are needed with a variety of digital and traditional 
interfaces toconfirm the extent to which features identified through this study are most salient to 
students’ learning experiences and to identify other salient features. 

Finally, given increasing concern with students’ ability to integrate knowledge across courses, the 
Adaptive Map toolitself could be used to investigate the effects of cross-course concept mapping. 
Expanding the tool’s content from a single course (statics) to closely related mechanics courses 
such as dynamics and mechanics of deformable bodies would allow not only the ability to study the 
effect of the tool with differing subject materials, but support cross course connections that could be 
used by students. As this study showed, students were more likely to explore relevant prior content 
with the non-linear layout in the Adaptive Map tool; cross-course connections would allow students 
to easily review relevant topics addressed in previous courses before addressing more advanced 
topics in later courses. In addition, it would also be interesting to explore why some students engage 
in Advance Organizer usage or prior content exploration while others do not. 

In summary, this research demonstrates the potential of concept map based navigation systems 
for digital content repositories and the value of collaboration between the engineering education 
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and information visualization communities to support the development of digital learning materi¬ 
als. The findings serve as the base for future research and development of such educational tools. 

The Adaptive Map Digital Textbook Project is an ongoing effort to provide a quality course re¬ 
source. The tool is designed to fill the role that traditional textbooks have previously filled in courses; 
serving as a trusted out-of-the-classroom reference for relevant information. Though features may 
evolve over time, the tool will always seek to fill this role. To access the latest version of the tool and 
for best practices, please visit the website at adaptivemap.ma.psu.edu . 
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